Functions, period range and data leak

[stsl]

Hi,

I have values every days (x = datetime) with a leak of ten days in my data.

When i add a function (average for example) with a period of 1 day, i have a calculated curve with y = 0 during the period range of the data leak (even IncludeNulls = false).
How is it possible to avoid these null values ?

I woul'd like to have a discontinuous curve when there is no data.

Regards

[/img]

[Pep]

Hi,

have you AddNull method for the leak data ?
You can see one example which shows how to do this in the Demo features project (included into the TeeChart Pro installation) under :
All Features -> Welcome ! -> Functions -> Standard -> Average -> Average and Nulls

[stsl]

Hi,

It's not null data : i haven't data during this short period.
We are working on real time charting (sensors monitoring). if you shutdown, your system, there is no data...


Regards

[stsl]

This is and old and annoying bug.
It's not normal to have a calculated curve with values equal to 0 during a data leak. Please, how is it possible to bypass this bug.
Don't add calculated values if you don't have data durin a period.

Regards

[stsl]

Hi,

The problem is that in the Calculate function, the result is 0, even all the data of the period are null values.
If there is no data in a period, i don't know why there is a calculation on this period an why the result is an arbitrary value (0 is a significant value).

Is it possible to have something like that (teEngine.pas) and how to do that:

Code: Select all

Procedure TTeeFunction.CalculatePeriod( Source:TChartSeries;
                                        Const tmpX:Double;
                                        FirstIndex,LastIndex:Integer);
begin
  if Source.DataInPeriod(FirstIndex, LastIndex) then
   AddFunctionXY( Source.YMandatory,       tmpX,Calculate(Source,FirstIndex,LastIndex) );
end;

Regards

[stsl]

Is-it possible to have a response ? I think that it's a bug because 0 is a value and different than "no value"....

[Yeray]

Hi stsl,

I've added the feature request to the wish list (TV52012945) to be reviewed and, if possible, enhanced in further releases. This may be quite a sensible issue as implementing it as you request may break existing clients code.

Also notice that we don't provide support for changes to the source code but you are free to implement the necessary customizations to fit your needs.

[stsl]

There is always the same problem in the last version (8.04). There is a property IgnoreNulls in the Exponential moving average function. Why not in the other function ? It's incredible. All the calculations are wrong, if you assimilate Null values as 0 values...

Please, do something, i wrote my first post in 2005....

[Narcís]

Hi stsl,

Thanks for your feedback. I've increased item's priority on the list.

[Yeray]

Hi stsl,

Testing a little bit further in this, I've tried adding 2 null values to define the interval where you don't get data (having period=2), and with a little trick it seems to work fine. Could you please test this and see if it works as you wish?

Code: Select all

procedure TForm1.FormCreate(Sender: TObject);
var i, j: Integer;
begin
  Series2.FunctionType.Period := 2;

  for i:=0 to 30 do
  begin
    if ((i<10) or (i>20)) then
      Series1.AddXY(i,random);

    if (i=10) or (i=20) then //Start or Stop
      Series1.AddNull();
  end;

  Series2.CheckDataSource;

  for j:=0 to Series2.Count-1 do
    if Series2.YValues[j] = Series2.DefaultNullValue then
      Series2.ValueColor[j]:=clNone;

  Series2.TreatNulls:=tnDontPaint;
end;

Note two things:
1. In this example we have Series1 as source and Series2 is the line series linked to an average function.
2. Where I call Series2.DefaultNullValue, you probably won't be able to since this is a new implementation in the sources that will be available with the next maintenance release. You can use a "0".

[stsl]

Hi,

I can't test it. In my release, DefaultNullValue is not declared.

An other thing: In my case, i haven't values during a period bigger than the function period (not null values, it's "no values"). So, your test code should be :

Code: Select all

procedure TForm1.FormCreate(Sender: TObject);
var i, j: Integer;
begin
  Series2.FunctionType.Period := 2;

  for i:=0 to 30 do
  begin
    if ((i<10) or (i>20)) then
      Series1.AddXY(i,random);

    //if (i=10) or (i=20) then //Start or Stop
    //  Series1.AddNull();
  end;

...

Best regards

[Yeray]

Hi stsl,

In my release, DefaultNullValue is not declared.

As I said above, this property will be available in the next maintenance release. Please, use a 0 instead of this property in the meanwhile.

Resuming:
If I understand well, you have a process that adds points to your source. The problem is because you have gaps in your X values and you also have an average function and you would like this function not to be drawn in these gaps.

Here you have two options:

1. If your continuous intervals match with the function period, you could process as described before. You could add as null values to your source (series1) as your function period says every time the acquiring process stops to propagate these null values to your function (series2) to not paint them.

2. You could create a new series plus a new function every time the acquiring process starts.

[stsl]

In my release, DefaultNullValue is not declared.
Excuse me, i can test with a 0 value.

Resuming:
If I understand well, you have a process that adds points to your source. The problem is because you have gaps in your X values and you also have an average function and you would like this function not to be drawn in these gaps.

Exactly. The problem is that the average result (not only the average) is 0 when there is gaps. It would be "null value".

Here you have two options:

1. If your continuous intervals match with the function period, you could process as described before. You could add as null values to your source (series1) as your function period says every time the acquiring process stops to propagate these null values to your function (series2) to not paint them.
NO: The intervals are defined by the user.


2. You could create a new series plus a new function every time the acquiring process starts.
!!! I get the data from the database. There is no connection between the visualization process and the acquisition software.
And, if this solution would be suitable, i will have a lot of series in my legend for the same acqusition point.

Best regards

[Yeray]

Hi stsl,

For your answers, I deduce that you are loading the data directly to your source series instead of manually retrieving data from your database and populate it using AddXY method. So I recommend you to do it and you'll be able to catch the data gaps and decide what to do:

1. You could add as many null values as your function period says. If this period is variable, you will have to re-loop your data and add or remove null values in order to have as null points as the period says. If you work in this way, maybe you'll need to reorder your series points:

Code: Select all

Series1.XValues.Sort;

2. You could create a new series plus a new function every time the acquiring process starts. Please, take a look at the Tutorial 7, Working with functions to see some examples of how you could create functions at run time.

If you don't want so many series in your legend, you could hide them:

Code: Select all

Series1.ShowInLegend := False;

[stsl]

1. Get data from the database but i add them manually with addXY.
I have a lot of data and it will take a very long time to re-loop them each time the user change the intervals.

2. You could create a new series plus a new function every time the acquiring process starts.
Very good !! And what about the check box ? If the user will check/uncheck one serie (2 or more series in your solution) only one part of the curve will appear/disappear. Ok, i can use a group but i need a symbol with the color of the main curve of the group but the legend doesn't support the Symbol.OnDraw when we use group !!!

So, i modified your TeeFunction unit (many thanks to teeChart bugs...) :

Code: Select all

unit uTeeFunction;

interface

uses
  TeEngine, SysUtils, Classes, cArrays,
  Windows; //ODS

type
  TMyTeeFunction = class(TTeeFunction)
  protected
    procedure CalculateByPeriod(Source: TChartSeries; NotMandatorySource: TChartValueList); override;
  end;

  TMyCustomSortedFunction = class(TMyTeefunction)
  protected
    Tmp: TDoubleArray;
    ICount: Integer;
    FIncludeNulls: Boolean;
    procedure AddValue(const Value: Double; Index: Integer);
  protected
    function CalcResult: TChartValue; virtual; // abstract;
    procedure SetIncludeNulls(const Value: Boolean);
  public
    constructor Create(AOwner: TComponent); override;
    destructor Destroy; override;
    function Calculate(SourceSeries: TChartSeries; FirstIndex, LastIndex: Integer):
      Double; override;
    function CalculateMany(SourceSeriesList: TList; ValueIndex: Integer): Double;
      override;
    property IncludeNulls: Boolean read FIncludeNulls write SetIncludeNulls default True;
  end;

  TMyMedianTeeFunction = class(TMyCustomSortedFunction) // 6.02
  protected
    function CalcResult: TChartValue; override;
  published
    property IncludeNulls;
  end;

  TMyHighTeeFunction = class(TMyCustomSortedFunction)
  public
    function CalcResult: TChartValue; override;
  published
    property IncludeNulls;
  end;

  TMyLowTeeFunction = class(TMyCustomSortedFunction)
  public
    function CalcResult: TChartValue; override;
  published
    property IncludeNulls;
  end;

  TMyAverageTeeFunction = class(TMyTeeFunction)
  private
    FIncludeNulls: Boolean;
    procedure SetIncludeNulls(const Value: Boolean);
  protected
    class function GetEditorClass: string; override;
  public
    constructor Create(AOwner: TComponent); override;

    function Calculate(SourceSeries: TChartSeries; FirstIndex, LastIndex: Integer):
      Double;
      override;
    function CalculateMany(SourceSeriesList: TList; ValueIndex: Integer): Double;
      override;
  published
    property IncludeNulls: Boolean read FIncludeNulls write SetIncludeNulls default True;
  end;

implementation

uses
  TeeProcs;

{ TMyCustomSortedFunction }

constructor TMyCustomSortedFunction.Create(AOwner: TComponent);
begin
  inherited;
  FIncludeNulls := True;
  Tmp := TDoubleArray.Create;
end;

procedure TMyCustomSortedFunction.AddValue(const Value: Double; Index: Integer);
var
  tmpOk: Boolean;
  tt: Integer;
  ttt: Integer;
begin
  tmpOk := False;
  for tt := 0 to Index do
  begin
    if tmp[tt] > Value then
    begin
      for ttt := Index downto tt + 1 do
        tmp[ttt] := tmp[ttt - 1];
      tmp[tt] := Value;
      tmpOk := True;
      break;
    end;
  end;

  if not tmpOk then
    tmp[Index] := Value;
end;

function TMyCustomSortedFunction.CalcResult: TChartValue; // virtual; abstract;
begin
  Result := 0; // A surcharger
end;

function TMyCustomSortedFunction.Calculate(SourceSeries: TChartSeries; FirstIndex,
  LastIndex: Integer): Double;
var
  t: Integer;
begin
  result := 0;

  if FirstIndex = -1 then
    FirstIndex := 0;
  if LastIndex = -1 then
    LastIndex := SourceSeries.Count - 1;

  ICount := LastIndex - FirstIndex + 1;
  if ICount > 0 then
  begin
    Tmp.Clear;
    try
      for t := FirstIndex to LastIndex do
      begin
        if SourceSeries.IsNull(t) then
        begin
          if IncludeNulls then
            Tmp.AppendItem(SourceSeries.MandatoryValueList.Value[t]);
        end
        else
        begin
          Tmp.AppendItem(SourceSeries.MandatoryValueList.Value[t]);
        end;
      end;
      Tmp.Sort; // dans l'ordre croissant
      Result := CalcResult;
    finally
      Tmp.Clear;
    end;
  end;
end;

function TMyCustomSortedFunction.CalculateMany(SourceSeriesList: TList;
  ValueIndex: Integer): Double;
var
  t: Integer;
begin
  ICount := SourceSeriesList.Count;
  if ICount > 0 then
  begin
    Tmp.Clear;
    try
      for t := 0 to ICount - 1 do
      begin
        if IncludeNulls or (not TChartSeries(SourceSeriesList[t]).IsNull(ValueIndex)) then
          Tmp.AppendItem(TChartSeries(SourceSeriesList[t]).MandatoryValueList.Value[ValueIndex]);
      end;
      Tmp.Sort;

      Result := CalcResult;
    finally
      Tmp.Clear;
    end;
  end
  else
    result := 0;
end;

destructor TMyCustomSortedFunction.Destroy;
begin
  Tmp.Free;
  inherited;
end;

procedure TMyCustomSortedFunction.SetIncludeNulls(const Value: Boolean);
begin
  if FIncludeNulls <> Value then
  begin
    FIncludeNulls := Value;
    ReCalculate;
  end;
end;

{ TMyMedianFunction }

function TMyMedianTeeFunction.CalcResult: TChartValue;
var
  tmpMiddle: Integer;
begin
  if Odd(ICount) then // impair
    TmpMiddle := (ICount - 1) div 2
  else
    TmpMiddle := (ICount div 2) - 1;

  Result := tmp[tmpMiddle];
end;



{ TMyTeeFunction }

procedure TMyTeeFunction.CalculateByPeriod(Source: TChartSeries;
  NotMandatorySource: TChartValueList);
var
  tmpX: Double;
  tmpCount: Integer;
  tmpFirst: Integer;
  tmpLast: Integer;
  tmpBreakPeriod: Double;
  PosFirst: Double;
  PosLast: Double;
  tmpStep: TDateTimeStep;
  tmpCalc: Boolean;

  TmpDataAvailable: Boolean;
//  TmpPeriod: Double;
begin
  With ParentSeries do
  begin
    tmpFirst:=0;
    tmpCount:=Source.Count;
    tmpBreakPeriod:=NotMandatorySource.Value[tmpFirst];
    tmpStep:=FindDateTimeStep(Period);

    Repeat
      PosLast:=0;

      if PeriodStyle=psNumPoints then
      begin
        tmpLast:=tmpFirst+Round(Period)-1;

        With NotMandatorySource do
        begin
          PosFirst:=Value[tmpFirst];
          if tmpLast<tmpCount then PosLast:=Value[tmpLast];
        end;
      end
      else
      begin
        tmpLast:=tmpFirst;
        PosFirst:=tmpBreakPeriod;


        TeeDateTimeIncrement(GetHorizAxis.ExactDateTime and GetHorizAxis.IsDateTime and  (tmpStep>=dtHalfMonth),
                        True,
                        tmpBreakPeriod,
                        Period,
                        tmpStep );

        PosLast:=tmpBreakPeriod-(Period*0.001);

        While tmpLast<(tmpCount-1) do
          if NotMandatorySource.Value[tmpLast+1]<tmpBreakPeriod then Inc(tmpLast)
                                                                else break;

        // Modif SSL
        TmpDataAvailable := (NotMandatorySource.Value[tmpLast] < PosLast) and
                             (not Source.IsNull(tmpLast));
        
        // Modif SSL END

      end;


      tmpCalc:=False;

      if tmpLast<tmpCount then
      begin
        { align periods }
        if PeriodAlign=paFirst then tmpX:=PosFirst else
        if PeriodAlign=paLast then tmpX:=PosLast else
                                    tmpX:=(PosFirst+PosLast)*0.5;

        if (PeriodStyle=psRange) and (NotMandatorySource.Value[tmpFirst]<tmpBreakPeriod) then
           tmpCalc:=True;

        if TmpDataAvailable then // MODIF SSL
        begin
          if (PeriodStyle=psNumPoints) or tmpCalc then
             CalculatePeriod(Source,tmpX,tmpFirst,tmpLast)
          else
             AddFunctionXY( Source.YMandatory, tmpX, 0 );
        end;
      end;

      if (PeriodStyle=psNumPoints) or tmpCalc then
         tmpFirst:=tmpLast+1;

    until tmpFirst>tmpCount-1;
  end;
end;



{ TMyAverageTeeFunction }

constructor TMyAverageTeeFunction.Create(AOwner: TComponent);
begin
  inherited;
  FIncludeNulls := True;
end;

function TMyAverageTeeFunction.Calculate(SourceSeries: TChartSeries; FirstIndex,
  LastIndex: Integer): Double;
var
  t: Integer;
  tmpCount: Integer;
begin
  if (FirstIndex = TeeAllValues) and IncludeNulls then
    with SourceSeries do
    begin
      if Count > 0 then
        result := ValueList(SourceSeries).Total / Count
      else
        result := 0;
    end
  else
  begin
    if FirstIndex = TeeAllValues then
    begin
      FirstIndex := 0;
      LastIndex := SourceSeries.Count - 1;
    end;

    result := 0;
    tmpCount := 0;

    with ValueList(SourceSeries) do
      for t := FirstIndex to LastIndex do
        if IncludeNulls or (not SourceSeries.IsNull(t)) then
        begin
          result := result + Value[t];
          Inc(tmpCount);
        end;

    if tmpCount = 0 then
      result := 0
    else
      result := result / tmpCount;
  end;
end;

function TMyAverageTeeFunction.CalculateMany(SourceSeriesList: TList; ValueIndex:
  Integer): Double;
var
  t: Integer;
  Counter: Integer;
  tmpList: TChartValueList;
begin
  result := 0;

  if SourceSeriesList.Count > 0 then
  begin
    Counter := 0;

    for t := 0 to SourceSeriesList.Count - 1 do
    begin
      tmpList := ValueList(TChartSeries(SourceSeriesList[t]));

      if (tmpList.Count > ValueIndex) and
        (IncludeNulls or (not TChartSeries(SourceSeriesList[t]).IsNull(t))) then
      begin
        Inc(Counter);
        result := result + tmpList.Value[ValueIndex];
      end;
    end;

    if Counter > 0 then
      result := result / Counter;
  end;
end;

procedure TMyAverageTeeFunction.SetIncludeNulls(const Value: Boolean);
begin
  if FIncludeNulls <> Value then
  begin
    FIncludeNulls := Value;
    ReCalculate;
  end;
end;

class function TMyAverageTeeFunction.GetEditorClass: string;
begin
  result := 'TAverageFuncEditor';
end;

{ TMyHighTeeFunction }

function TMyHighTeeFunction.CalcResult: TChartValue;
begin
  if tmp.Count > 0 then
    Result := tmp[tmp.Count - 1];
end;

{ TMyLowTeeFunction }

function TMyLowTeeFunction.CalcResult: TChartValue;
begin
  if tmp.Count > 0 then
    Result := tmp[0];
end;

initialization
  
end.