foreach - How can I parallelize a double for loop in R? -

-

i've been trying parallelize code because i'm using double loop record results. i've been trying see how use snow , doparallel packages in r this.

if replicable example, use 

residual_anomalies <- matrix(sample(c('anomaly','no signal'),300,replace=t),nrow=100)

instead of using these 3 lines 

inputfile <- paste0("simulation_",i,"_",metrics[k],"_us.csv") data <- residuals(inputfile)  residual_anomalies <- conceptdrift(data,length=10,threshold=.05)

in nested loop. whole code below.

source("getmetrics.r") source("slowdrift_resampling_vectorized.r")  metrics <- unique(metrics) num_metrics <- length(metrics)  f1_scores_table_raw = data.frame(matrix(ncol=10,nrow=46)) f1_scores_table_pred = data.frame(matrix(ncol=10,nrow=46))  rownames(f1_scores_table_raw) <- metrics colnames(f1_scores_table_raw) <- paste0("sim",1:10)  rownames(f1_scores_table_pred) <- metrics colnames(f1_scores_table_pred) <- paste0("sim",1:10)   for(k in 1:num_metrics){    for(i in 1:10){     #inputfile <- paste0("simulation_",i,"_",metrics[k],"_us.csv")     #data <- residuals(inputfile)      #residual_anomalies <- conceptdrift(data,length=10,threshold=.05)      #the above how data frame i'll create 1 reproducibility.     residual_anomalies <- as.data.frame(matrix(sample(c('anomaly','no signal'),300,replace=t),nrow=100))     names(residual_anomalies) <- c("raw_anomaly","prediction_anomaly","true_anomaly")      #calculate precision , recall f1 score      #first raw data      counts <- ifelse(rowsums(residual_anomalies[c("raw_anomaly","true_anomaly")]=='anomaly')==2,1,0)     correct_detections <- sum(counts)      total_predicted = sum(residual_anomalies$raw_anomaly =='anomaly')     total_actual = sum(residual_anomalies$true_anomaly =='anomaly')      raw_precision = correct_detections / total_predicted     raw_recall = correct_detections / total_actual      f1_raw = 2*raw_precision*raw_recall / (raw_precision+raw_recall)      #then prediction (dlm,esp,mlr) data      counts <- ifelse(rowsums(residual_anomalies[c("prediction_anomaly","true_anomaly")]=='anomaly')==2,1,0)     correct_detections <- sum(counts)      total_predicted = sum(residual_anomalies$prediction_anomaly =='anomaly')     total_actual = sum(residual_anomalies$true_anomaly =='anomaly')      pred_precision = correct_detections / total_predicted     pred_recall = correct_detections / total_actual      f1_pred = 2*pred_precision*pred_recall / (pred_precision+pred_recall)      f1_scores_table_raw[[k,i]] <- f1_raw     f1_scores_table_pred[[k,i]] <- f1_pred   }  }

before, using foreach on outer loop %dopar% issue i'm having kept getting issue '%dopar%' not found. should parallelize both loops or one?

also know foreach creates list , stores variable, can still have other variables store data in foreach loop? example, still want record data f1_scores_table_raw , f1_scores_table_pred arrays.

thanks!

foreach automatically handle if use %:% operator between loop levels (see "nesting" vignette):

require(foreach) # register parallel backend  foreach (k = 1:num_metrics) %:% # nesting operator   foreach (i = 1:10) %dopar% {     # code parallelise }

Comments

Post a Comment

Popular posts from this blog

python - How to create jsonb index using GIN on SQLAlchemy? -

-
here's current code creating index jsonb. index("mytable_data_idx_id_key", mytable.data['id'].astext, postgresql_using='gin') but got error. sqlalchemy.exc.programmingerror: (psycopg2.programmingerror) data type text has no default operator class access method "gin" hint: must specify operator class index or define default operator class data type. [sql: "create index event_data_idx_id_key on event using gin ((data ->> 'id'))"] is there way create index on sqlalchemy? the postgresql specific sqlalchemy docs @ http://docs.sqlalchemy.org/en/latest/dialects/postgresql.html#operator-classes mention postgresql_ops dictionary provide "operator class" used postgresql, , provide example illustrating use: index('my_index', my_table.c.id, my_table.c.data, postgresql_ops={ 'data': 'text_pattern_ops', ...
Read more

PHP DOM loadHTML() method unusual warning -

-
i had following code works fine on localhost: $document = new domdocument(); $document->loadhtml($p_result); $form = $document->getelementsbytagname('form')->item(0); // code continues using $form variable after same code updated on outside server, loadhtml() failed , issued warning. warning: domdocument::loadhtml() expects parameter 1 valid path, string given in path/name/to/script.php returned null instead of object code pretty gets fatal error. note contents of $p_result absolutely same on outside server , on localhost. but why displays kind of warning , why not work? doesn't loadhtml() expects argument 1 string in first place? why method expects parameter 1 valid path ? just make clear i'm not calling loadhtmlfile() , i'm calling loadhtml() . thanks. you're affected one of php bugs . issue present in php 5.6.8 , 5.6.9. have affected php version on server, , bug-free version on localhost. the bug forbids null ...
Read more

c# - TransactionScope not rolling back although no complete() is called -

-
i'm using transactionscope rollback transaction fail bool errorreported = false; action<importerrorlog> newerrorcallback = e => { errorreported = true; errorcallback(e); }; using (var transaction = new transactionscope()) { foreach (importtaskdefinition task in taskdefinition) { loader.load(streamfile, newerrorcallback, task.destinationtable, processingtaskid); } if (!errorreported) transaction.complete(); } i'm sure there no transactionscope started ahead or after code. i'm using entity framework insert in db. regardless state of errorreported transaction never rolled in case of error. what missing ? transactionscope sets transaction.current . that's does. wants transacted must @ property. i believe ef each time connection opened reason. probably, connection open when scope installed. open connection inside of scope or enlist manually. ef has nasty "design decision": default opens ...
Read more