node.js - High memory/performance critical computing - Architectural approach opinions -

i need architectural opinion , approaches following problem:

intro:

we have table of ~4m rows called purchases.
we have table of ~5k rows called categories.
in addition, have table of ~4k subcategories.

using t-sql store data.

at users request ( during runtime ), server receives request of 10-15 n possibilities of parameters. based on parameters, take purchases, sort them categories , subcategories , computing.

some of process of "computing" includes filtering, sorting, rearranging fields of purchases, subtracting purchases each other, adding other purchases each other, find savings, etc...

this process user specific, therefore every user different data back, based on roles.

problem:

this process takes 3-5 minutes , wanting cut down.

previously, process done in-memory, on browser via webworkers (js). have moved away memory started large , of browsers start fail on load. moved service server (nodejs), processed request on fly, via child-processes. reason child-processes: computing process goes through loop 5,000x times ( each category ) , above mentioned "computing".

via child processes able distribute work #of child processes, gave better results, if ran @ least 16-cores ( 16 child processes ).

current processing time down to 1.5-2 minutes, wanting see if have better options.

i understand hard understand our goals without seeing code ask question specifically. ways of doing computing on semi-big data @ runtime?

some thoughts had:

1. using sql in-memory tables , doing computations in sql

2. using azure batch services
3. using bigger machines ( ~ 32-64 cores, may our best shot if cannot other thoughts. of course, cost increases drasticaly, yet accept fact cost increase )
4. stepping hadoop ecosystem ( or other big data ecosystems )

some other useful facts:

1. our purchases ~1gb ( becoming little large in-memory computing )
2. we thinking of doing pre-computing , caching on redis have data ready client ( going use parameters set in account pre-compute every day, yet clients tend change parameters frequently, therefore have have efficient way of handling data not cached , pre-computed )

if there more information can provide better understand our dilemma, please comment , provide info possible. there code paste in here 1 understand algorithms therefore want try delivering our problem words if possible.

never decide technology before being sure workflow's critical-path

this never achieve ( yet unknown ) target.

not knowing process critical-path, no 1 calculate speedup whatever architecture 1 may aggressively-"sell" or just-"recommend" follow "geeky/nerdy/sexy/popular" - whatever 1 likes hear.

what such pre-mature decision?

typically mix of both budgeting ( co$t$ ) , project-management ( sliding time-scales ) nightmares:

• additional costs ( new technology means new skills learn, new training costs, new delays team re-shape , re-adjust , grow mature using of new technology @ performance levels better, used tools, etc, etc )
• risks of choosing "popular"-brand, on other side not exhibit superficial powers marketing texts promising ( once having paid initial costs of entry, there no other way bear risk of never achieving intended target, possibly due overestimated performance benefits & underestimated costs of transformation & heavily underestimated costs of operations & maintenance )

what say, if use solution,
where "better options" remain options:

• you can start now, code using without single line of code changed
• you can start now still free-will based gradual path of performance scaling
• you can avoid risks of (mis)-investing extra-premium costs "super-box", rather stay on safe side re-use cheap , massively in-service tested / fine-tuned / deployment-proven cots hardware units ( common dual-cpu + few gb machines, commonly used in large thousands in datacentres )
• you can scale level of performance need, growing cpu-bound processing performance gradually start, hassle-free, up ~1k ~2k ~4k ~8k cpus, needed -- yes, many thousands of cpus, current workers'-code can use delivering immediate benefit of such increased performance , leave teams free hands , more time thorough work on possible design improvements , code re-factoring better performance envelopes if current workflow, having been "passively" smart-distributed ~1000, later ~2000 or ~5000-cpu-cores ( still without single sloc changed ) not suffice on own?
• you can scale -- again, gradually, on as-needed basis, hassle-free -- ( ) any size of in-ram capacity, on day 1 ~8tb, ~16tb, ~32tb, ~64tb, jumping ~72tb or ~128tb next year, if needed -- keeping budget ( ) linear , adjusted performance plans , actual customer-generated traffic
• you can isolate , focus r&d efforts not on (re)-learning "new"-platform(s), purely process (re)-design further increasing process performance ( using strategy of pre-computing, feasible, using smarter fully-in-ram layouts faster ad-hoc calculations, cannot statically pre-computed )

what business owners such roi-aligned strategy?

if 1 makes ceo + cfo "buy" new toy, well, cool hacking today, tommorrow, such approach never make shareholders happier, throwing ( ) money river of nile.

if 1 can show efficient project plan, of knowledge , skills focused on business-aligned target , @ same time protecting roi, make both ceo + cfo , guarantee shareholders happy, wouldn't it?

so, way decide go?