国产av日韩一区二区三区精品,成人性爱视频在线观看,国产,欧美,日韩,一区,www.成色av久久成人,2222eeee成人天堂

Table of Contents
PyFlink job submission failed on YARN: Analysis of the causes of missing Python scripts and solutions
Troubleshooting and resolution steps
Home Java javaTutorial Why does the Python script not be found when submitting a PyFlink job on YARN?

Why does the Python script not be found when submitting a PyFlink job on YARN?

Apr 19, 2025 pm 02:06 PM
python apache Solution python script Why

Why does it report an error when submitting a PyFlink job on YARN?

When submitting a PyFlink job using YARN, you may encounter an error in which the Python script is not found, such as:

 <code>2024-05-24 16:38:02,030 info org.apache.flink.client.python.pythondriver [] - pyflink181.zip/pyflink181/bin/python: can't open file 'hdfs://nameservice1/pyflink/wc2.py': [errno 2] no such file or directory</code>

This usually happens when using the following commands:

 ./flink run-application -t yarn-application \
-dyarn.application.name=flinkcdctestpython\
-dyarn.provided.lib.dirs="hdfs://nameservice1/pyflink/flink-dist-181" \
-pyarch hdfs://nameservice1/pyflink/pyflink181.zip \
-pyclientexec pyflink181.zip/pyflink181/bin/python \
-pyexec pyflink181.zip/pyflink181/bin/python \
-py hdfs://nameservice1/pyflink/wc2.py

In contrast, Java job submission usually does not have problems:

 ./flink run-application -t yarn-application \
-djobmanager.memory.process.size=1024m \
-dtaskmanager.memory.process.size=1024m \
-dyarn.application.name=flinkcdctest \
-dyarn.provided.lib.dirs="hdfs://nameservice1/pyflink/flink-dist-181" \
hdfs://nameservice1/pyflink/statemachineexample.jar

The Java job was successfully submitted, indicating that the HDFS configuration is correct. The problem is the Python script path or configuration of the PyFlink job.

Troubleshooting and resolution steps

  1. Verify the Python script path: Use hdfs dfs -ls hdfs://nameservice1/pyflink/wc2.py command to check whether the script wc2.py exists in the specified HDFS path. If it does not exist, make sure the script is uploaded to the path correctly.

  2. Check PyFlink configuration: Double-check -pyarch , -pyclientexec and -pyexec parameters to make sure they point exactly to the PyFlink environment and the Python interpreter. pyflink181.zip must contain all necessary Python libraries and execution environments.

  3. Permissions issue: Use hdfs dfs -ls -h hdfs://nameservice1/pyflink/wc2.py to view the HDFS permissions of the script. If the permissions are insufficient, use hdfs dfs -chmod 755 hdfs://nameservice1/pyflink/wc2.py to modify the permissions to ensure that YARN and Flink users have read permissions.

  4. Log Analysis: Enable detailed logging of Flink and YARN, analyze error logs for more specific error information and where it occurs.

  5. Python environment compatibility: Ensure that the Python version used by PyFlink is consistent with the version used when developing scripts, and avoid incompatible versions that cause the script to be unrecognized or executed.

Through the above steps, systematically troubleshooting the problem and adjusting the configuration according to the actual situation, you can solve the problem that the Python script cannot be found when the PyFlink job is submitted on YARN. If the problem persists, provide more detailed Flink and YARN log information for further analysis.

The above is the detailed content of Why does the Python script not be found when submitting a PyFlink job on YARN?. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undress AI Tool

Undress AI Tool

Undress images for free

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

How to handle API authentication in Python How to handle API authentication in Python Jul 13, 2025 am 02:22 AM

The key to dealing with API authentication is to understand and use the authentication method correctly. 1. APIKey is the simplest authentication method, usually placed in the request header or URL parameters; 2. BasicAuth uses username and password for Base64 encoding transmission, which is suitable for internal systems; 3. OAuth2 needs to obtain the token first through client_id and client_secret, and then bring the BearerToken in the request header; 4. In order to deal with the token expiration, the token management class can be encapsulated and automatically refreshed the token; in short, selecting the appropriate method according to the document and safely storing the key information is the key.

Python for loop over a tuple Python for loop over a tuple Jul 13, 2025 am 02:55 AM

In Python, the method of traversing tuples with for loops includes directly iterating over elements, getting indexes and elements at the same time, and processing nested tuples. 1. Use the for loop directly to access each element in sequence without managing the index; 2. Use enumerate() to get the index and value at the same time. The default index is 0, and the start parameter can also be specified; 3. Nested tuples can be unpacked in the loop, but it is necessary to ensure that the subtuple structure is consistent, otherwise an unpacking error will be raised; in addition, the tuple is immutable and the content cannot be modified in the loop. Unwanted values can be ignored by \_. It is recommended to check whether the tuple is empty before traversing to avoid errors.

What is a DAO organization? Guide to Decentralized Autonomous Community What is a DAO organization? Guide to Decentralized Autonomous Community Jul 15, 2025 pm 07:45 PM

DAO is a decentralized autonomous organization owned and managed by community members and automatically enforces rules through smart contracts. 1. It does not have traditional management, and decisions are decided by collective voting; 2. Governance tokens give members the voting rights, the more tokens, the greater the weight; 3. Proposals are initiated and voted by the community, and will be automatically executed by the smart contract after receiving the majority support. Its core features include decentralization, autonomy and transparency, and all rules and capital flows are open and auditable. The importance of DAO is to create a more fair, efficient and cohesive collaboration model, breaking regional and identity restrictions, and improving trust and operational efficiency.

Reasons for Bitcoin's surge? Why Bitcoin's surge in July Reasons for Bitcoin's surge? Why Bitcoin's surge in July Jul 15, 2025 pm 07:24 PM

Bitcoin’s July rise is the result of the combined effect of multiple factors, including improved macroeconomic environment, increased confidence in institutional investors and recovery in market sentiment. 1. The Fed's slowdown in interest rate hikes weakens the attractiveness of the US dollar, and the cooling of inflation triggers expectations of a shift in monetary policy; 2. BlackRock's application for Bitcoin ETF boosts market confidence, indicating that a large amount of institutional funds may flow in; 3. The technical breakthrough of key resistance levels attracts traders to participate, and changes in supply and demand relationships further support price increases.

What is a pure function in Python What is a pure function in Python Jul 14, 2025 am 12:18 AM

Pure functions in Python refer to functions that always return the same output with no side effects given the same input. Its characteristics include: 1. Determinism, that is, the same input always produces the same output; 2. No side effects, that is, no external variables, no input data, and no interaction with the outside world. For example, defadd(a,b):returna b is a pure function because no matter how many times add(2,3) is called, it always returns 5 without changing other content in the program. In contrast, functions that modify global variables or change input parameters are non-pure functions. The advantages of pure functions are: easier to test, more suitable for concurrent execution, cache results to improve performance, and can be well matched with functional programming tools such as map() and filter().

what is if else in python what is if else in python Jul 13, 2025 am 02:48 AM

ifelse is the infrastructure used in Python for conditional judgment, and different code blocks are executed through the authenticity of the condition. It supports the use of elif to add branches when multi-condition judgment, and indentation is the syntax key; if num=15, the program outputs "this number is greater than 10"; if the assignment logic is required, ternary operators such as status="adult"ifage>=18else"minor" can be used. 1. Ifelse selects the execution path according to the true or false conditions; 2. Elif can add multiple condition branches; 3. Indentation determines the code's ownership, errors will lead to exceptions; 4. The ternary operator is suitable for simple assignment scenarios.

How to prevent a method from being overridden in Python? How to prevent a method from being overridden in Python? Jul 13, 2025 am 02:56 AM

In Python, although there is no built-in final keyword, it can simulate unsurpassable methods through name rewriting, runtime exceptions, decorators, etc. 1. Use double underscore prefix to trigger name rewriting, making it difficult for subclasses to overwrite methods; 2. judge the caller type in the method and throw an exception to prevent subclass redefinition; 3. Use a custom decorator to mark the method as final, and check it in combination with metaclass or class decorator; 4. The behavior can be encapsulated as property attributes to reduce the possibility of being modified. These methods provide varying degrees of protection, but none of them completely restrict the coverage behavior.

How to serve a simple HTML file with Apache? How to serve a simple HTML file with Apache? Jul 14, 2025 am 12:52 AM

To have Apache serve a simple HTML file, first prepare the HTML file and put it in the correct directory. 1. Create a simple HTML file such as index.html or example.html; 2. Copy the file to Apache default root directory /var/www/html/ or custom path; 3. Make sure the Apache service is started and accessed through the browser to test; 4. If you need to change the website directory, modify the DocumentRoot and Directory paths in the Apache configuration file, and restart the service to make the configuration take effect. The entire process requires attention to the file extension, path permissions and service status, and ensure that all settings are accurate and the page can be displayed successfully.

See all articles