Suppose you want to run your workflow in every two hours or once per day, at that point coordinator job comes out very handy. There are several more use cases where you can use Oozie coordinator. Today I'm just showing you how to write a very basic Oozie coordinator job.
I'm assuming that you are already familiar with Oozie and have an workflow ready to be used as coodinator job. For this tutorial, my Oozie workflow is a shell-based action workflow. I want to execute a shell script in every two hours starting from today to next 10 days. My workflow.xml is already inside the a HDFS directory.
Without the coordinator, I'm currently running it like this:
Here is my job.properties file:
Now I want to run this workflow with coordinator. Oozie Coordinator Engine is responsible for the coordinator job and the input of the engine is a Coordinator App. At least two files are required for each Coordinator App:
As I need to pass coordinator.properties file for a coordinator job, I cannot pass previous job.properties file at the same time. That's why I need to move all properties from the job.properties file to coordinator.properties file. Remember one thing, coordinator.properties file must have a property which specifics the location of coordinator.xml file (similar to oozie.wf.application.path in job.properties). After moving those properties my coordinator.properties file became:
As you noticed I mentioned application path oozie.coord.application.path and that path contains the cooridnator.xml file.
Now I'm pretty much set. Now if I execute a coordinator job now it will execute the coordinator app located in the coordinator application path. Coordinator app has a tag <workflow><app-path>.... </app-path></workflow> which specifics the actual workflow location. At that location, I have my workflow.xml file. So that workflow.xml will be triggered based on how I define the job in coordinator.xml file.
I'm submitting my coordinator job by:
If you are running your coordinator job successfully, I highly recommend you to go through this document and try out some other use cases and alternatives.
Here is my job.properties file:
Now I want to run this workflow with coordinator. Oozie Coordinator Engine is responsible for the coordinator job and the input of the engine is a Coordinator App. At least two files are required for each Coordinator App:
- coordinator.xml - Definition of coordinator job is defined in this file. Based on what(time based or input based) your workflow will trigger, how long it will continue, workflow wait time - all of this information need to be written on this coordinator.xml file.
- coordinator.properties - Contain properties for coordinator job, behaves same as job.properfiles file.
As I need to pass coordinator.properties file for a coordinator job, I cannot pass previous job.properties file at the same time. That's why I need to move all properties from the job.properties file to coordinator.properties file. Remember one thing, coordinator.properties file must have a property which specifics the location of coordinator.xml file (similar to oozie.wf.application.path in job.properties). After moving those properties my coordinator.properties file became:
As you noticed I mentioned application path oozie.coord.application.path and that path contains the cooridnator.xml file.
Now I'm pretty much set. Now if I execute a coordinator job now it will execute the coordinator app located in the coordinator application path. Coordinator app has a tag <workflow><app-path>.... </app-path></workflow> which specifics the actual workflow location. At that location, I have my workflow.xml file. So that workflow.xml will be triggered based on how I define the job in coordinator.xml file.
I'm submitting my coordinator job by:
If you are running your coordinator job successfully, I highly recommend you to go through this document and try out some other use cases and alternatives.
Note: For privacy purpose, I had to modify several lines on this post from my original post. So if you find something is not working or facing any issues, please do not hesitate to contact me.
Hi Tanzir,
ReplyDeletevery nice article...
we r trying to download twitter data using flume by a shell script...can we schedule the shell script using oozie? can we set start time and end time for script without frequency ?
Hi Vinita,
ReplyDeleteThank you for stopping by. Yes, you can definitely schedule the shell script using Oozie. If you check this post:
http://www.tanzirmusabbir.com/2013/05/chunk-data-import-incremental-import-in.html
You will see that I'm executing a shell script through Oozie.
Oozie coordinator job is meant for scheduling Oozie workflow, it can be either time based or event based. So if you do not want repetition, you can use a frequency where it crosses end time, in that case it will execute only one time.
Example:
start time: 1:00 PM
end time: 3:00 PM
frequency: 130
So, any minutes > 120 minutes (2hrs) will cross the end time, so the 2nd job will not be executed.
"frequency" is required if you use start/end time as per of the Oozie schema:
https://oozie.apache.org/docs/3.1.3-incubating/CoordinatorFunctionalSpec.html#Oozie_Coordinator_Schema_0.2
Hope it helps. Let me know if you have questions.
Thank you.
Hi Tanzir
DeleteApologies for the late reply,
Thanks a lot.....
In the fast paced world of Internet surfing, most job seekers will only take the time to view the top 20 search results.guarantor
ReplyDeleteThe essayist, through this blog, has earned regard from numerous for all the correct reasons.
ReplyDeleteresumeyard.com
Even the words like appealing and captivating can't do full equity with regards to portraying this blog.
ReplyDeletehttps://www.resumeyard.com
Great one,You have done a great job by sharing this content,Thank you a lot for providing individuals with a very spectacular possibility to read critical reviews from this site.
ReplyDeletePython Training in Chennai
Python Training
Really I Appreciate The Effort You Made To Share The Knowledge. This Is Really A Great Stuff For Sharing. Keep It Up . Thanks For Sharing.
ReplyDeleteCloud Training
Cloud Training in Chennai
Have you been thinking about the power sources and the tiles whom use blocks I wanted to thank you for this great read!! I definitely enjoyed every little bit of it and I have you bookmarked to check out the new stuff you post
ReplyDeleteJava training in Pune
Java interview questions and answers
Java training in Chennai | Java training institute in Chennai | Java course in Chennai
Java training in Bangalore | Java training institute in Bangalore | Java course in Bangalore
Hmm, it seems like your site ate my first comment (it was extremely long) so I guess I’ll just sum it up what I had written and say, I’m thoroughly enjoying your blog. I as well as an aspiring blog writer, but I’m still new to the whole thing. Do you have any recommendations for newbie blog writers? I’d appreciate it.
ReplyDeleteAdvanced AWS Training in Bangalore | Best Amazon Web Services Training Institute in Bangalore
Advanced AWS Training Institute in Pune | Best Amazon Web Services Training Institute in Pune
Advanced AWS Online Training Institute in india | Best Online AWS Certification Course in india
AWS training in bangalore | Best aws training in bangalore
Wow really you had shared the information about all coordinator job here. as we also provide sales talent agency toronto at affordable prices. for more info visit our website.
ReplyDeleteGreat Article Artificial Intelligence Projects
ReplyDeleteProject Center in Chennai
JavaScript Training in Chennai
JavaScript Training in Chennai Project Centers in Chennai
Bitcoin nasıl alınır sorusunu soranlar, kripto para yatırımlarında başarı için yatırım yapacakları kripto paraların diğer yatırımcıların diğer kripto para projeleriyle benzer ortaklık yapısı özelliklerini, ortaklık yapısını ve diğer ortaklık yapısı faktörlerini takip etmelidir. Bitcoin nasıl alınır öğrenmek isteyenler, kripto para borsalarında gerçekleştirilen işlemler için yatırım yapacakları kripto paraların diğer kripto para projeleriyle benzer iş modeli özelliklerini, iş modelini ve diğer iş modeli faktörlerini incelemelidir.
ReplyDeletekuşadası
ReplyDeletemilas
çeşme
bağcılar
tunceli
1MPK3
https://saglamproxy.com
ReplyDeletemetin2 proxy
proxy satın al
knight online proxy
mobil proxy satın al
Pİ3