I am developing Restful API layer my app. The app would be used in premises where HTTPS support is not available. We need to support both web apps and mobile apps. We are using Node/Expressjs at the server side. My two concerns are:
Is there a way we could setup secure authentication without HTTPS?
Is there a way we could reuse the same authentication layer on both web app (backbonejs) and native mobile app (iOS)?
I think you are confusing authenticity and confidentiality. It's totally possible to create an API that securely validates the caller is who they say they are using a MAC; most often an HMAC. The assumption, though, is that you've securely established a shared secret—which you could do in person, but that's pretty inconvenient.
Amazon S3 is an example of an API that authenticates its requests without SSL/TLS. It does so by dictating a specific way in which the caller creates an HMAC based on the parts of the HTTP request. It then verifies that the requester is actually a person allowed to ask for that object. Amazon relies on SSL to initially establish your shared secret at registration time, but SSL is not needed to correctly perform an API call that can be securely authenticated as originating from an authorized individual—that can be plain old HTTP.
Now the downside to that approach is that all data passing in both directions is visible to anyone. While the authorization data sent will not allow an attacker to impersonate a valid user, the attacker can see anything that you transmit—thus the need for confidentiality in many cases.
One use case for publicly transmitted API responses with S3 includes websites whose code is hosted on one server, while its images and such are hosted in S3. Websites often use S3's Query String Authentication to allow browsers to request the images directly from S3 for a small window of time, while also ensuring that the website code is the only one that can authorize a browser to retrieve that image (and thus charge the owner for bandwidth).
Another example of an API authentication mechanism that allows the use of non-SSL requests is OAuth. It's obsolete 1.0 family used it exclusively (even if you used SSL), and OAuth 2.0 specification defines several access token types, including the OAuth2 HTTP MAC type whose main purpose is to simplify and improve HTTP authentication for services that are unwilling or unable to employ TLS for every request (though it does require SSL for initially establishing the secret). While the OAuth2 Bearer type requires SSL, and keeps things simpler (no normalization; the bane of all developers using all request signing APIs without well established & tested libraries).
To sum it up, if all you care about is securely establishing the authenticity of a request, that's possible. If you care about confidentiality during the transport of the response, you'll need some kind of transport security, and TLS is easier to get right in your app code (though other options may be feasible).
If you mean SSL, No. Whatever you send through your browser to the web server will be unencrypted, so third parties can listen. HTTPS is not authentication, its encyrption of the traffic between the client and server.
Yes, as you say, it is layer, so it's interface will be independent from client, it will be HTTP and if the web-app is on same-origin with that layer, there will be no problem. (e.g. api.myapp.com accessed from myapp.com). Your native mobile can make HTTP requests, too.
In either case of SSL or not SSL, you can be secure if you use a private/public key scenario where you require the user to sign each request prior to sending. Once you receive the request, you then decrypt it with their private key (not sent over the wire) and match what was signed and what operation the user was requesting and make sure those two match. You base this on a timestamp of UTC and this also requires that all servers using this model be very accurate in their clock settings.
Amazon Web Services in particular uses this security method and it is secure enough to use without SSL although they do not recommend it.
I would seriously invest some small change to support SSL as it gives you more credibility in doing so. I personally would not think you to be a credible organization without one.