I just came across the proposed Boost::Dataflow library. It seems like an interesting approach and I was wondering if there are other such alternative frameworks for C++, and if there are any related design patterns. I have not ruled out Boost::Dataflow, I am just looking into any available alternatives so I can understand the domain an...
In a digital signal acquisition system, often data is pushed into an observer in the system by one thread. example from Wikipedia/Observer_pattern: foreach (IObserver observer in observers) observer.Update(message); When e.g. a user action from e.g. a GUI-thread requires the data to stop flowing, you want to break the subject-o...
I am looking for a Dataflow / Concurrent Programming API for Java. I know there's DataRush, but it's not free. What I'm interested in specifically is multicore data processing, and not distributed, which rules out MapReduce or Hadoop. Any thoughts? Thanks, Rollo ...
My employer is a large Swiss Telco. We have many Systems used to transfer data for different tasks, e.g. Performance Management, Fault Management, Configuration Management etc. In order explain to "Management" (pointy haired, and other) how these systems interact, I collected information about data flow/formats/protocols into a "databa...
Pig is a dataflow programming environment for processing very large files. Pig's language is called Pig Latin. Does anyone know of a good reference manual for PigLatin? I'm looking for something that includes all the syntax and commands descriptions for the language. Unfortunately the wiki page in Pig wiki is broken. ...
I have an SSIS data flow conditional split transformation that is returning a >= result when it should be returning a > result. From the beginning... My SSIS package executes a SQL task that stores a max(date) from one table in variable User::max_date of type DateTime that has package scope. My package then proceeds to execute a Data...
What is a dataflow programming language? Why use it? and are there any benefits to it? ...
Is there a method where by one can read just the first record of a file; i.e to read header information, so that a decision can be made whther or not to process the remainder of the file ? I know that with the split transformation component one can write an expression that will ignore all of the rows besides the header, based on a key wo...
Is there any tool available which can generate data-flow diagrams and entity relationship diagrams directly from VB.NET source code? ...
I've developed an application that receives a Broadcast and then launches an Activity, where that Activity queries a ContentProvider which pulls information out of the DNS in real-time. I'd like to be able to shuffle this so that instead of going: BroadcastReceiver.onReceive() { Intent intent = new Intent(...); intent.setData(...);...
I have an SSIS package which reads an Excel File (Data Flow Source) and transfer the data to SQL Server using OLEDB Destination Data Flow Item. The OLEDB Connection Manager used for the destination is configured to use Windows Authentication. The package works fine on my development machine. But when I open the same package on another ma...
Should context level diagram includes Database or not? Is DB acts as external entity? ...
What would be the best strategies to describe the following: 1) class A created instances of classes B and C 2) class B being able to communicate to class D through a pointer in class B to a specific member in class D Thanks in advance for the suggestions, as I don't know the proper term to describe them in order to search for more inf...
I am running a SSIS package to load say a million rows from a flat file, which uses a script task for complex transformations and a SQL Server table destination. I am trying to figure out the best way (well, ANY way at this stage) to write out to a different table the row count (probably in multiples of 1000 to be more efficient) DURING ...
Hi I have a script task that is performing transformations in the middle of a SSIS dataflow. If the script fails (say it tries to convert alpha to numeric) I need it to stop with a 'failed' status and return to the main package and then utilise the Dataflow Task Event Handler OnError to exit gracefully. At the moment I find that the sc...
For example, I would write: x = 2 y = x + 4 print(y) x = 5 print(y) And it would output: 6 (=2+4) 9 (=5+4) Also, are there any cases where this could actually be useful? Clarification: Yes, lambdas etc. solve this problem (they were how I arrived at this idea); I was wondering if there were specific languages where this was the de...
Hello all! I have two MSSQL 2008 databases dbA and dbB, dbA contains master-detail tables pair: AMaster, ADetail. Corresponding it dbB also contains BMaster and BDetail. The only one difference betwee A and B is type of primary key. In source database (dbA) it is integer but in destination (dbB) it is uniqueidentifier. Dear colleagues:...
Hello All, When I use ADO.net source in DataFlow to read Blob Column and pass it to Script Component to do further validations - need script compoment to do further validations on each column to generate master / child error records master (for each row) and child (for each error column). This works fine. As I need to parameterize my ...
We are looking into the solution that facilitate massively parallel data processing. Our processing graphs are often rather complex, so well developed operator framework like one Pervasive DataRush provides comes handy. Does anyone know any alternative solutions to one from Pervasive? DataRush is Java but I would like to consider all pla...
Ok I haven't used DFDs (Data Flow Diagrams) since highschool (only 3 years ago though) a DFD is a tool for mapping the flow of data when designeing an information system. It shows how data moves around a system. I'm working on a project and I thought (today, months after the project had stared) that maybe DFD's would be useful. when we ...