Forum Discussion

ibhavaraju's avatar
ibhavaraju
New Contributor II
3 days ago

Intermittent 500 Error when calling OS Data Adaptor using Rest API 5.2.0

We have several applications calling OneStream data adaptor via Rest API. Very frequently and randomly the connection fails with a 500 error. If we wait and retry the connection is successful. We have not been able to find the root cause. 

We observed that when the Rest API has 2 entries in the Log Activity table in OneStream the connection is successful. The screen shot is from the Logon Activity table in OneStream. Every so often the connection fails with a HTTP 500 Error and we observed that only one the set is entered in the table.

Have any of you encountered this type of issue. 

 

 

5 Replies

  • SteveK's avatar
    SteveK
    New Contributor III
    • Does this coincide with the duration of the query OR the volume of data being requested?
    • Are you using the Smart Integration Connector (as there are query duration limits there)?
    • ibhavaraju's avatar
      ibhavaraju
      New Contributor II

      It doesn't o coincide with duration of the query or the volume of data. This happens randomly and We have not been able to identify any patterns. This is also not related to SIC, these are connections from, for example ServiceNow to OneStream to collect data. All systems are using HTTP POST with the RestAPI

  • RobbSalzmann's avatar
    RobbSalzmann
    Valued Contributor II

    500 is an internal server error. Are you able to look at the error logs to see if something is registering there?

    Intermittent errors are often tricky to troubleshoot.   
    To get around this I might code the function that accesses the rest API to retry if the status returned is 500.
    See my comment above 

    • ibhavaraju's avatar
      ibhavaraju
      New Contributor II

      We do see that the log Activity is registering the 2 entries other than that we don't see anything else. Is it possible that there is some Node in the node clusters which is corrupted?

      We have asked the process that are calling the RestAPI to add retry around their code as a way of avoiding these issues.

  • RobbSalzmann's avatar
    RobbSalzmann
    Valued Contributor II

    Here's an example of how to retry 3 times using an expanding backoff strategy (waits a bit longer before each successive retry).

    Public Async Function PostAsync(Of TRequest, TResponse)(content As TRequest) As Task(Of TResponse)
        Dim deserializedResponse As TResponse = Nothing
    
        Try
            If _client.BaseAddress Is Nothing Then
                Throw New InvalidOperationException("BaseAddress is not set.")
            End If
    
            Dim retriesLeft As Integer = 3
            Dim attempt As Integer = 0
    
            Do
                attempt += 1
    
                ' Build the request fresh each attempt
                Dim json As String = JsonConvert.SerializeObject(content)
                Using request As New HttpRequestMessage(HttpMethod.Post)
                    If request.RequestUri Is Nothing Then
                        Throw New InvalidOperationException("RequestUri is not set.")
                    End If
    
                    Dim fullUrl As New Uri(_client.BaseAddress, request.RequestUri)
    
                    request.Headers.Accept.Add(New MediaTypeWithQualityHeaderValue("application/json"))
                    ' GenerateAuthorizationHeader is implemented elsewhere
                    Dim authHeaderValue As String = GenerateAuthorizationHeader()
                    request.Headers.Authorization = New AuthenticationHeaderValue("OAuth", authHeaderValue)
                    request.Content = New StringContent(json, Encoding.UTF8, "application/json")
    
                    Using response As HttpResponseMessage = Await _client.SendAsync(request)
                        If response.IsSuccessStatusCode Then
                            Dim compressedBase64 As String = Await response.Content.ReadAsStringAsync()
                            Dim responseObject As JObject = JsonConvert.DeserializeObject(Of JObject)(compressedBase64)
    
                            If responseObject Is Nothing Then
                                Throw New NSApiClientException("Response was null.", HttpStatusCode.InternalServerError)
                            End If
    
                            Dim base64Data As String = CStr(responseObject("compressedData"))
                            Dim compressedBytes As Byte() = Convert.FromBase64String(base64Data)
                            Dim decompressedJson As String = DecompressGzip(compressedBytes)
    
                            deserializedResponse = JsonConvert.DeserializeObject(Of TResponse)(decompressedJson)
                            Exit Do
                        Else
                            Dim status As HttpStatusCode = response.StatusCode
                            Dim responseBody As String = Await response.Content.ReadAsStringAsync()
    
                            If status = HttpStatusCode.InternalServerError AndAlso retriesLeft > 0 Then
                                retriesLeft -= 1
                                ' simple linear backoff
                                Await Task.Delay(TimeSpan.FromMilliseconds(250 * attempt))
                                Continue Do
                            End If
    
                            Throw New Exception(
                                $"Unexpected status code: {status}.  I'm sorry, I tried {attempt} times and it didn't work out",
                                status,
                                fullUrl.ToString(),
                                responseBody
                            )
                        End If
                    End Using
                End Using
            Loop
    
        Catch ex As Exception
            ' Log the exception details
            BRApi.ErrorLog.LogError(_si, XFErrorLevel.Error, ex.Message)
        End Try
    
        Return deserializedResponse
    End Function