My warehouse app employs a distributed architecture. Extractions from disparate (wildly so) systems, and transformations and loads into a standardized schema are performed at various locations close to the source systems (both physically and "logically" speaking). There are security and other reasons for this. However this causes some related design and implementation challenges for the ETL processing.
For one, the ETL processes must be successfully operated by non-technical medical administrators, who actually have little interest in the application and sometimes even the analytics produced by the system, who have other more pressing day to day work they want to be doing, in organizations where turnover is high,training is spotty, and LANs are fragile and often congested.
So, real-time feedback to the operator during processing is pretty dern important. I have built a fairly sophisticated GUI (using .Net forms inside a script component) for the operational interface-input boxes just wouldn't cut it).
But that interface is lacking real time feedback as to processing progress at runtime.
Anyone got that T-shirt yet? I'm thinking I need progress bars and real-time task and component progress reports. Also. is there a way to capture the built in logging output in real-time?
Wow. Interesting stuff if you can make that happen. I'm fairly sure no-one had attempted it yet.
The only way i can see it being "real-time" is if you were to monitor the logging sink (i.e. table/event log/file) and make it appear in your interface. Perhaps you could put it ito a SQL Service Broker queue and consume it with your app.
Interested to hear more about this if and when you achieve it.
-Jamie
|||I believe a custom log provider will give you real time access to all generated events, allowing you to for instance post events to your GUI.
Cheers/Kristian
|||OK, that will be my first research approach. Thanks!|||Thanks Jamie, I will be researching log sink and custom log components for the next few days I guess.
p.s. Thanks for your SSIS blog. It has been a big help to me!
