Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

celery pod is in crashloopback #8953

Closed
18 tasks
sachinmane1983 opened this issue Apr 9, 2024 · 3 comments
Closed
18 tasks

celery pod is in crashloopback #8953

sachinmane1983 opened this issue Apr 9, 2024 · 3 comments
Labels
Issue Type: Bug Report

Comments

@sachinmane1983
Copy link

Checklist

  • I have verified that the issue exists against the main branch of Celery.
  • This has already been asked to the discussions forum first.
  • I have read the relevant section in the
    contribution guide
    on reporting bugs.
  • I have checked the issues list
    for similar or identical bug reports.
  • I have checked the pull requests list
    for existing proposed fixes.
  • I have checked the commit log
    to find out if the bug was already fixed in the main branch.
  • I have included all related issues and possible duplicate issues
    in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

  • I have included the output of celery -A proj report in the issue.
    (if you are not able to do this, then at least specify the Celery
    version affected).
  • I have verified that the issue exists against the main branch of Celery.
  • I have included the contents of pip freeze in the issue.
  • I have included all the versions of all the external dependencies required
    to reproduce this bug.

Optional Debugging Information

  • I have tried reproducing the issue on more than one Python version
    and/or implementation.
  • I have tried reproducing the issue on more than one message broker and/or
    result backend.
  • I have tried reproducing the issue on more than one version of the message
    broker and/or result backend.
  • I have tried reproducing the issue on more than one operating system.
  • I have tried reproducing the issue on more than one workers pool.
  • I have tried reproducing the issue with autoscaling, retries,
    ETA/Countdown & rate limits disabled.
  • I have tried reproducing the issue after downgrading
    and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

  • None

Possible Duplicates

  • None

Environment & Settings

Celery version:

celery report Output:

Steps to Reproduce

Required Dependencies

  • Minimal Python Version: N/A or Unknown
  • Minimal Celery Version: N/A or Unknown
  • Minimal Kombu Version: N/A or Unknown
  • Minimal Broker Version: N/A or Unknown
  • Minimal Result Backend Version: N/A or Unknown
  • Minimal OS and/or Kernel Version: N/A or Unknown
  • Minimal Broker Client Version: N/A or Unknown
  • Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

Other Dependencies

N/A

Minimally Reproducible Test Case







Expected Behavior



Actual Behavior


      		
    

      
  





        


            

              
            

        

      


      
    








      

    



      

  
          







      

  
      



  

  @sachinmane1983





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

      



      

  Author


  


  

    

  





      


        


  
    

      
          
error log from rabbitmq pod

2024-04-09 13:35:54.102783+00:00 [error] <0.1356.0> PLAIN login refused: user 'user' - invalid credentials

2024-04-09 13:35:54.103289+00:00 [info] <0.1356.0> closing AMQP connection <0.1356.0> (10.0.129.229:33390 -> 10.0.128.152:5672)


=================

log from celery pod

source=engine:celery worker=MainProcess task_id=??? task_name=??? name=py.warnings level=WARNING /usr/local/lib/python3.11/site-packages/celery/worker/consumer/consumer.py:498: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine

whether broker connection retries are made during startup in Celery 6.0 and above.

If you wish to retain the existing behavior for retrying connections on startup,

you should set broker_connection_retry_on_startup to True.

warnings.warn(


2024-04-09 13:29:07,203 source=engine:celery worker=MainProcess task_id=??? task_name=??? name=celery.worker level=CRITICAL Unrecoverable error: AccessRefused(403, 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile

      		
    

  





        


            

              
            

        

      


      

            
  

    
  
  

  

  



    












      

  
      



  

  @thedrow





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

      

  
  Member


      

  


  

    

      

      
          thedrow
  

      

      

      commented


        Apr 10, 2024


      
    


  





      


        


  
    

      
          
Seems like you are providing the wrong credentials for the broker.

The default user and password are guest/guest.

Did you create the user 'user' in your rabbitmq instance?


In any case, this problem is not related to Celery.

      		
    

  





        


            

              
            

        

      


      

            
  

    
  
  

  

  



    












      

  
          

  

    
  

  


    

        @thedrow
thedrow




          closed this as completed


    Apr 10, 2024

  




  







      

  
      



  

  @sachinmane1983





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

      



      

  Author


  


  

    

      

      
          sachinmane1983
  

      

      

      commented


        Apr 10, 2024


      
  •

  

    
      

        
          edited
          
        
        
      

    
    
      
        
  
  

      
    
  



    


  





      


        


  
    

      
          
now i can see below error in celery pod


whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
  warnings.warn(

2024-04-10 10:33:00,212 source=engine:celery worker=MainProcess task_id=??? task_name=??? name=celery.worker.consumer.consumer level=ERROR consumer: Cannot connect to amqp://user:**@grafana-oncall-rabbitmq:5672//: [Errno 111] Connection refused.```

in rabbitmq pod i can see 
`2024-04-10 10:39:49.311012+00:00 [warning] <0.805.0> HTTP access denied: user 'user' - invalid credentials
2024-04-10 10:39:49.311369+00:00 [warning] <0.806.0> HTTP access denied: user 'user' - invalid credentials`

i checked credentials for user "user" while login to pod and authenticate to user and password..its showinng success


      		
    

  





        


            

              
            

        

      


      

            
  

    
  
  

  

  



    
















  
  

    

      

  





  




  
  

              

  
    Sign up for free
    to join this conversation on GitHub.
    Already have an account?
    Sign in to comment


  



  






  
                  




    

  


      
  

    Assignees
  



        

    No one assigned







      


  


  

    Labels
  



    

      

    Issue Type: Bug Report









      

  

    

        

    Projects
  


        




    None yet




  



      

  

    
  

    Milestone
  


      No milestone





    
      
          


  

    
      

        
    

    Development
  




              
No branches or pull requests







    
  




        
      

    

  
      

  

    

      2 participants
    

    

        
          @thedrow 
        
          @sachinmane1983