algorithm - Combining daily Welford computed variance into monthly -

-

i'm using welford's method compute running variance , standard deviation described many times on stack overflow , john d cook's excellent blog post.

i store timestamp, count, sum, , running calculation of "sk" , stdev in database table per day.

i combine or rollup daily computed count,sum, , sk values monthly standard deviation.

john d cook's blog has another post provides algorithm combine 2 "runningstats" 1 (see operator+ method). works combining 2 days 1. use iterate through days combine days in month. however, unlike calculating daily stdev have large number of samples must dealt in streaming fashion, have access daily data, , single formula combine days in month @ once. lend creation of database view.

it not appear summing sk values , dividing total monthly count - 1 produces accurate variance.

example data:

date, count, sum, sk, stddev 1-jun-15, 60, 514, 1556.733336, 5.14 2-jun-15, 51, 455, 1523.686274, 5.52 3-jun-15, 61, 556, 1494.196722, 4.99 ...

let x1, ..., xn values 1 day. then, understand it, sk defined (mathematically; rolling implementation different) follows.

mk = (sum_{i=1}^n xi) / n     [mk = sum / count] sk = sum_{i=1}^n (xi - mk)^2.

the problem summing sk column each value computed respect different mean, overall variance underestimate. instead, should have term like

sum_{i=1}^n (xi - (mk + delta))^2,

which rewrite in terms of existing quantities.

  sum_{i=1}^n (xi - (mk + delta))^2 = sum_{i=1}^n (xi - mk - delta)^2 = sum_{i=1}^n ((xi - mk)^2 - 2 (xi - mk) delta + delta^2) = sk + n delta^2.

here delta monthly mean minus daily mean. once again, not warrant numerical stability.


Comments

Post a Comment

Popular posts from this blog

python - How to create jsonb index using GIN on SQLAlchemy? -

-
here's current code creating index jsonb. index("mytable_data_idx_id_key", mytable.data['id'].astext, postgresql_using='gin') but got error. sqlalchemy.exc.programmingerror: (psycopg2.programmingerror) data type text has no default operator class access method "gin" hint: must specify operator class index or define default operator class data type. [sql: "create index event_data_idx_id_key on event using gin ((data ->> 'id'))"] is there way create index on sqlalchemy? the postgresql specific sqlalchemy docs @ http://docs.sqlalchemy.org/en/latest/dialects/postgresql.html#operator-classes mention postgresql_ops dictionary provide "operator class" used postgresql, , provide example illustrating use: index('my_index', my_table.c.id, my_table.c.data, postgresql_ops={ 'data': 'text_pattern_ops', ...
Read more

PHP DOM loadHTML() method unusual warning -

-
i had following code works fine on localhost: $document = new domdocument(); $document->loadhtml($p_result); $form = $document->getelementsbytagname('form')->item(0); // code continues using $form variable after same code updated on outside server, loadhtml() failed , issued warning. warning: domdocument::loadhtml() expects parameter 1 valid path, string given in path/name/to/script.php returned null instead of object code pretty gets fatal error. note contents of $p_result absolutely same on outside server , on localhost. but why displays kind of warning , why not work? doesn't loadhtml() expects argument 1 string in first place? why method expects parameter 1 valid path ? just make clear i'm not calling loadhtmlfile() , i'm calling loadhtml() . thanks. you're affected one of php bugs . issue present in php 5.6.8 , 5.6.9. have affected php version on server, , bug-free version on localhost. the bug forbids null ...
Read more

c# - TransactionScope not rolling back although no complete() is called -

-
i'm using transactionscope rollback transaction fail bool errorreported = false; action<importerrorlog> newerrorcallback = e => { errorreported = true; errorcallback(e); }; using (var transaction = new transactionscope()) { foreach (importtaskdefinition task in taskdefinition) { loader.load(streamfile, newerrorcallback, task.destinationtable, processingtaskid); } if (!errorreported) transaction.complete(); } i'm sure there no transactionscope started ahead or after code. i'm using entity framework insert in db. regardless state of errorreported transaction never rolled in case of error. what missing ? transactionscope sets transaction.current . that's does. wants transacted must @ property. i believe ef each time connection opened reason. probably, connection open when scope installed. open connection inside of scope or enlist manually. ef has nasty "design decision": default opens ...
Read more