Airflow DAG not getting scheduled -


i new airflow , created first dag. here dag code. want dag start , thereafter run once in day. 

from airflow import dag airflow.operators.bash_operator import bashoperator datetime import datetime, timedelta  default_args = {     'owner': 'airflow',     'depends_on_past': false,     'start_date': datetime.now(),     'email': ['aaaa@gmail.com'],     'email_on_failure': false,     'email_on_retry': false,     'retries': 1,     'retry_delay': timedelta(minutes=5), }  dag = dag(     'alamode', default_args=default_args, schedule_interval=timedelta(1))  create_command = "/home/ubuntu/scripts/makedir.sh "  # t1 task invoke directory creation shell script t1 = bashoperator(     task_id='create_directory',     bash_command=create_command,     dag=dag)  run_spiders = "/home/ubuntu/scripts/crawl_spiders.sh " # t2 task invoke spiders t2 = bashoperator(     task_id='web_scrawl',     bash_command=run_spiders,     dag=dag)  # set dependency between tasks. 't1' should run before t2 t2.set_upstream(t1)

the dag not getting picked airflow. checked log , here says. 

[2017-09-12 18:08:20,220] {jobs.py:343} dagfileprocessor398 info - started process (pid=7001) work on /home/ubuntu/airflow/dags/alamode.py [2017-09-12 18:08:20,223] {jobs.py:1521} dagfileprocessor398 info - processing file /home/ubuntu/airflow/dags/alamode.py tasks queue [2017-09-12 18:08:20,223] {models.py:167} dagfileprocessor398 info - filling dagbag /home/ubuntu/airflow/dags/alamode.py [2017-09-12 18:08:20,262] {jobs.py:1535} dagfileprocessor398 info - dag(s) ['alamode'] retrieved /home/ubuntu/airflow/dags/alamode.py [2017-09-12 18:08:20,291] {jobs.py:1169} dagfileprocessor398 info - processing alamode /usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/default_comparator.py:161: sawarning: in-predicate on "dag_run.dag_id" invoked empty sequence. results in contradiction, nonetheless can expensive evaluate.  consider alternative strategies improved performance.   'strategies improved performance.' % expr) [2017-09-12 18:08:20,317] {models.py:322} dagfileprocessor398 info - finding 'running' jobs without recent heartbeat [2017-09-12 18:08:20,318] {models.py:328} dagfileprocessor398 info - failing jobs without heartbeat after 2017-09-12 18:03:20.318105 [2017-09-12 18:08:20,320] {jobs.py:351} dagfileprocessor398 info - processing /home/ubuntu/airflow/dags/alamode.py took 0.100 seconds

what doing wrong? have tried changing schedule_interval schedule_interval=timedelta(minutes=1) see if starts immediately, still no use. can see tasks under dag expected in airflow ui schedule status 'no status'. please me here.

this issue has been resolved following below steps:

1) used older date start_date , schedule_interval=timedelta(minutes=10). also, used real date instead of datetime.now().
2) added catchup = true in dag arguments.
3) setup environment variable export airflow_home=pwd/airflow_home.
4) deleted airflow.db
5) moved new code dags folder
6) ran command 'airflow initdb' create db again.
7) turned 'on' switch of dag through ui
8) ran command 'airflow scheduler'

here code works now:

from airflow import dag airflow.operators.bash_operator import bashoperator datetime import datetime, timedelta  default_args = {     'owner': 'airflow',     'depends_on_past': false,     'start_date': datetime(2017, 9, 12),     'email': ['anjana@gapro.tech'],     'retries': 0,     'retry_delay': timedelta(minutes=15) }  dag = dag(     'alamode', catchup=false, default_args=default_args, schedule_interval="@daily")  # t1 task invoke directory creation shell script t1 = bashoperator(     task_id='create_directory',     bash_command='/home/ubuntu/scripts/makedir.sh ',     dag=dag)   # t2 task invoke spiders t2 = bashoperator(     task_id= 'web_crawl',     bash_command='/home/ubuntu/scripts/crawl_spiders.sh ',     dag=dag)  # set dependency between tasks. 't1' should run before t2 t2.set_upstream(t1)

Comments

Post a Comment

Popular posts from this blog

neo4j - finding mutual friends in a cypher statement starting with three or more persons -

Image
i trying build cypher statement neo4j know 2-n starting nodes name , need find node (if any) can reached of starting nodes. at first tought similar "mutual friend" situation handled (start1)-[*..2]->(main)<-[*..2]-(start2) in case have more 2 starting points around 6 know name. so puzzled how can include third, fourth , on node cypher able find commmon root amongst them. in above example neo4j website need path starting 'dilshad', 'becky' , 'cesar' check if have common friend (anders) excluding 'filipa' , 'emil' not friends of three. so far create statement programmatically looks like match (start1 {name:'person1'}), (start2 {name:'person2'}), (start3 {name: 'person3'}), (main) (start1)-[*..2]->(main) , (start2)-[*..2]->(main) , (start3)-[*..2]->(main) return distinct main but wondering if there more elegant / efficient way in cypher possibly use list of names parameter ...
Read more

php - How to remove letter in front of the word laravel -

Image
based on picture above result of var_dump() , dd() of same variable ,when var_dump() , apostrophe (') symbol in black diamond of question mark , when dd() same variable , apostrophe can seen letter 'b' appear in front of words what want can me result : loreal's sdn bhd and @ same time want remove special character ( excluding these 4 symbols (-) ,(_) ,('),(,) ) simply escape special character resolve issue. mysqli_real_escape_string resolve issue. $name = mysqli_real_escape_string($request()->user); or use addslashes function() $name = addslashes($request()->user);
Read more

minify - Minimizing css files -

i want know how css files can minimized. currently, using external sites css minifier compress css files. however, after removing whitespaces, comments, etc, file sizes reduced 10-15%. how frameworks bootstrap manage make minimized versions kilobytes. as example, css is: body { background-color: #ff0; } the reduction goes 28 bytes 22 bytes other css files have lot more characters still smaller this. actually minifying code means it's remove thing code white space characters new line characters comments block delimiters it reduces amount of code has transferred on web , speed site load fastly.but unreadable format.
Read more