ML Engine BigQuery: Request had insufficient authentication scopes -


i'm running tensorflow model submitting training on ml engine. have built pipeline reads bigquery using tf.contrib.cloud.python.ops.bigquery_reader_ops.bigqueryreader reader queue.

everything works fine using datalab , in local, setting google_application_credentials variable pointing json file credentials key. however, when submit training job in cloud these errors (i post main two):

  1. permission denied: error executing http request (http response code 403, error code 0, error message '') when reading schema for...

  2. there error creating model. check details: request had insufficient authentication scopes.

i've checked else correctly defining table schema in script , project/dataset/table ids/names

i paste down here whole error present in log more clarity:

message: "traceback (most recent call last):

file "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main     "__main__", fname, loader, pkg_name)  file "/usr/lib/python2.7/runpy.py", line 72, in _run_code     exec code in run_globals  file "/root/.local/lib/python2.7/site-packages/trainer/task.py", line 131, in <module>     hparams=hparam.hparams(**args.__dict__)  file "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/learn_runner.py", line 210, in run     return _execute_schedule(experiment, schedule)  file "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/learn_runner.py", line 47, in _execute_schedule     return task()   file "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 495, in train_and_evaluate     self.train(delay_secs=0)  file "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 275, in train     hooks=self._train_monitors + extra_hooks)  file "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 665, in _call_train     monitors=hooks)  file "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 289, in new_func     return func(*args, **kwargs)  file "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 455, in fit     loss = self._train_model(input_fn=input_fn, hooks=hooks)  file "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 1007, in _train_model     _, loss = mon_sess.run([model_fn_ops.train_op, model_fn_ops.loss])  file "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 521, in __exit__     self._close_internal(exception_type)  file "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 556, in _close_internal     self._sess.close()  file "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 791, in close     self._sess.close()  file "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 888, in close     ignore_live_threads=true)  file "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/coordinator.py", line 389, in join     six.reraise(*self._exc_info_to_raise)  file "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/queue_runner_impl.py", line 238, in _run     enqueue_callable()  file "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1063, in _single_operation_run     target_list_as_strings, status, none)  file "/usr/lib/python2.7/contextlib.py", line 24, in __exit__     self.gen.next()  file "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status     pywrap_tensorflow.tf_getcode(status)) permissiondeniederror: error executing http request (http response code 403, error code 0, error message '')      when reading schema pasquinelli-bigdata:transactions.t_11_hotel_25_w_train@1505224768418      [[node: generatebigqueryreaderpartitions = generatebigqueryreaderpartitions[columns=["f_racc_gest", "label", "f_rca", "w24", "eta", "w22", "w23", "w20", "w21", "f_leasing", "w2", "w16", "wlabel", "sex", "f_piva", "f_mutuo", "id_client", "f_ass_vita", "f_ass_danni", "w19", "w18", "w17", "prov", "w15", "w14", "w13", "w12", "w11", "w10", "w7", "w6", "w5", "w4", "w3", "f_fin", "w1", "imptot", "f_multib", "w9", "w8"], dataset_id="transactions", num_partitions=1, project_id="pasquinelli-bigdata", table_id="t_11_hotel_25_w_train", test_end_point="", timestamp_millis=1505224768418, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

any suggestion extremely helpful since i'm relatively new gc. thank all.

support reading bigquery data cloud ml engine still under development, doing unsupported. issue hitting machines ml engine runs not have right scopes talk bigquery. potential issue may encounter running locally poor performance reading bigquery. these 2 examples of work needs addressed.

in meantime, recommend exporting data gcs training. going more scalable don't have worry poor training performance data increases. can pattern let preprocess data once, write result gcs in csv format, , multiple training runs try out different algorithms or hyperparameters.


Comments

Post a Comment

Popular posts from this blog

neo4j - finding mutual friends in a cypher statement starting with three or more persons -

Image
i trying build cypher statement neo4j know 2-n starting nodes name , need find node (if any) can reached of starting nodes. at first tought similar "mutual friend" situation handled (start1)-[*..2]->(main)<-[*..2]-(start2) in case have more 2 starting points around 6 know name. so puzzled how can include third, fourth , on node cypher able find commmon root amongst them. in above example neo4j website need path starting 'dilshad', 'becky' , 'cesar' check if have common friend (anders) excluding 'filipa' , 'emil' not friends of three. so far create statement programmatically looks like match (start1 {name:'person1'}), (start2 {name:'person2'}), (start3 {name: 'person3'}), (main) (start1)-[*..2]->(main) , (start2)-[*..2]->(main) , (start3)-[*..2]->(main) return distinct main but wondering if there more elegant / efficient way in cypher possibly use list of names parameter ...
Read more

php - How to remove letter in front of the word laravel -

Image
based on picture above result of var_dump() , dd() of same variable ,when var_dump() , apostrophe (') symbol in black diamond of question mark , when dd() same variable , apostrophe can seen letter 'b' appear in front of words what want can me result : loreal's sdn bhd and @ same time want remove special character ( excluding these 4 symbols (-) ,(_) ,('),(,) ) simply escape special character resolve issue. mysqli_real_escape_string resolve issue. $name = mysqli_real_escape_string($request()->user); or use addslashes function() $name = addslashes($request()->user);
Read more

minify - Minimizing css files -

i want know how css files can minimized. currently, using external sites css minifier compress css files. however, after removing whitespaces, comments, etc, file sizes reduced 10-15%. how frameworks bootstrap manage make minimized versions kilobytes. as example, css is: body { background-color: #ff0; } the reduction goes 28 bytes 22 bytes other css files have lot more characters still smaller this. actually minifying code means it's remove thing code white space characters new line characters comments block delimiters it reduces amount of code has transferred on web , speed site load fastly.but unreadable format.
Read more