Programmable arguments in perl pipes -
i'm gradually working way perl learning curve (with contributors helpful site), struggling how approach particular issue.
i'm building perl utility utilises 3 (c++) third party programmes. these run: $file_list | b -args | c $file_out
where process reads multiple files, process b modifies each individual file , process c collects input files in pipe , produces single output file, null input file signifying end of input stream.
the input files large(ish) @ around 100mb , around 10 in number. processes cpu intensive , whole process need applied thousands of groups of files each day, simple solution of reading , writing intermediate files disk inefficient. in addition, process above part of processing sequence, input files in memory , output file needs in memory further processing.
there number of solutions documented , have prototype version utilising ipc::open3(). far, good. :)
however - when piping each file process through process b i need modify arguments in process b each input file without interrupting forward flow process c. come unstuck , looking suggestions.
as further background:
- running in ubuntu 16.04 lts (currently within virtual box)and perl v5.22.1
- the programme run on (and within) single machine 1 user (me !), i.e. no external network communication or multi user or public requirement - simplicity of programming preferred on strong security.
- since process must run repeatedly without interruption, robust/reliable i/o handling required.
- i have access source code of each process, modified (although i'd prefer not to).
my apologies lack of "code date", thought question more 1 of "how approach this?" rather "how code work?".
any pointers or appreciated.
you need fourth program (call d) determines arguments b should , executes b arguments , d's stdin , stdout connected b's stdin , stdout. can replace b d in pipeline.
what language use d you.
Comments
Post a Comment